[WIP, NO MERGE]: Added recurring job to sync automation runs to Tinybird - #30472
Closed
troyciesco wants to merge 1 commit into
Closed
[WIP, NO MERGE]: Added recurring job to sync automation runs to Tinybird#30472troyciesco wants to merge 1 commit into
troyciesco wants to merge 1 commit into
Conversation
ref https://linear.app/ghost/issue/NY-1559/ghost-backend-send-events-to-traffic-analytics-service-with-recurring This cannot be merged as-is because it includes database migrations, so it will need to be broken up further before review. - Replaces the boot.js proof of concept with a tinybird-sync service scheduled through the jobs service every five minutes, with a per-process random offset - Adds a tinybird_syncs table storing a per-table watermark (last_synced_updated_at), plus updated_at indexes on automation_runs and automation_run_steps so each run only scans recently changed rows - Sends only the columns the Tinybird materialized views read; member id, member email, and lock bookkeeping never leave MySQL - Reads rows in keyset-paginated batches and splits requests by UTF-8 byte size under Tinybird's Events API limit - Uses wait=true so Tinybird acknowledges committed rows, fails the job on quarantined rows or HTTP errors without advancing the watermark, and aborts stalled requests after a timeout - Holds back rows updated within the last minute so a still-open transaction cannot be skipped by the watermark - Syncs both tables concurrently and guards against overlapping runs - Adds a backgroundJobs.tinybirdSync config flag to disable the job - Reverts the WIP changes that read stats from Tinybird; that work belongs to NY-1560 - Covers the sync logic with unit tests against an in-memory SQLite database and a fake Tinybird Express server
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueComment |
github-actions
Bot
added
the
migration
Contributor
|
It looks like this PR contains a migration 👀 General requirements
Schema changes
Data changes
|
|
| Command | Status | Duration | Result |
|---|---|---|---|
nx run ghost:test:ci:integration |
❌ Failed | 4m 20s | View ↗ |
nx run ghost:test:integration |
❌ Failed | 2m 24s | View ↗ |
nx run @tryghost/admin:test:acceptance |
✅ Succeeded | 7m 59s | View ↗ |
nx run ghost:test:ci:e2e |
✅ Succeeded | 3m 59s | View ↗ |
nx run ghost:test:legacy |
✅ Succeeded | 3m 9s | View ↗ |
nx run ghost:test:e2e |
✅ Succeeded | 2m 8s | View ↗ |
nx run ghost-monorepo:lint:boundaries |
✅ Succeeded | 24s | View ↗ |
nx run-many -t test:unit -p @tryghost/admin,@tr... |
✅ Succeeded | 42s | View ↗ |
Additional runs (6) |
✅ Succeeded | ... | View ↗ |
💡 Dealing with memory or CPU issues? See memory and CPU details with the resource usage add-on ↗.
☁️ Nx Cloud last updated this comment at 2026-09-02 19:56:41 UTC
troyciesco
commented
Sep 7, 2026
| module.exports = addTable('tinybird_syncs', { | ||
| id: { type: 'string', maxlength: 24, nullable: false, primary: true }, | ||
| table_name: { type: 'string', maxlength: 191, nullable: false, unique: true }, | ||
| last_synced_updated_at: { type: 'dateTime', nullable: false }, |
Contributor
Author
There was a problem hiding this comment.
note to self: consider adding last_synced_id here as well. i just ran into an issue when rebooting locally where it tried to rerun everything, because i'd created the runs/steps with the data generator so they all had the same updated_at as last_synced_updated_at.
not sure if this is the answer and/or if we shouldn't do >=, just a reminder to look at it
Contributor
|
Obsoleted by this PR stack: #30567 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ref https://linear.app/ghost/issue/NY-1559/ghost-backend-send-events-to-traffic-analytics-service-with-recurring
This cannot be merged as-is because it includes database migrations, so it will need to be broken up further before review.
This probably will actually replace its parent pr #30441 instead of getting merged into it, but i wanted to take it one step at a time and leave the stack we were working from intact for now.